Run federated searches over remote Splunk platform deployments
Federated Search for Splunk lets you search across specific datasets on multiple Splunk platform deployments. These deployments include your local Splunk platform deployment and remote Splunk platform deployments that you designate as federated providers. A federated search is processed partly on your local instance and partly on the federated providers. The remote and local results are then brought to the federated search head of your local deployment, where they are aggregated together to produce a final result set.
The federated search process includes search optimizations that help federated searches run as efficiently as possible. These optimizations filter results on the remote search heads, which reduces the amount of data that must be sent over the network, and then process the results on your federated search head, improving overall search performance.
The experience of writing and running federated searches differs depending on whether your federated providers use standard mode or transparent mode.
For an overview of federated search and federated search terminology, see About Federated Search for Splunk.
General requirements for federated search
Before you can run a federated search, your administrator must designate one or more remote deployments as federated providers. See Define a Splunk platform federated provider. Find out whether the providers you are searching over use standard mode or transparent mode.
Federated provider names added to search results
When you run federated searches, the names of the federated providers you search are added to your search results as the values of a field named splunk_federated_provider
. You can use the splunk_federated_provider
field to group or filter results by the federated providers that produced them.
For example, if you run transparent mode federated searches over multiple federated providers, you can use the splunk_federated_provider
field to discern how many results are coming from each provider. Without this field it can be hard to tell where your results are coming from. The following transparent mode federated search breaks out result counts by provider name and then again by product name.
sourcetype=access_* | stats count(eval(splunk_federated_provider="ProviderWest")) AS Provider_West, count(eval(splunk_federated_provider="ProviderEast")) AS Provider_East BY productName
This feature applies to all federated searches, whether they are in standard mode or transparent mode.
This feature requires that your local Splunk platform deployment and all remote deployments set up as federated providers be upgraded to the following Splunk platform versions:
- Splunk Cloud Platform 9.0.2305 or higher.
- Splunk Enterprise 9.0.6 or higher Splunk Enterprise 9.0.x versions.
- Splunk Enterprise 9.1.1 or higher.
Search over a standard mode federated provider
Federated searches that run over a standard mode federated provider must use a special search syntax to invoke the federated search. When you work with standard mode federated providers and you run a search that does not include this syntax, the search runs only over your local deployment.
Standard mode federated searches are subject to specific requirements and restrictions.
Additional requirements for standard mode federated search
- Your administrator must create one or more federated indexes on your local federated search head. Each federated index maps to a specific remote dataset on a federated provider. See Define a Splunk platform federated provider.
- Your role must have permissions for the federated indexes on your local federated search head that you intend to search. See the section on federated indexes in Service accounts and security for Federated Search for Splunk.
- Any custom knowledge objects in your search such as lookups, calculated fields, or event types must be present on the local search head and the remote search head. If this duplication of knowledge objects is not present or is incorrectly applied, searches may fail or return incorrect results. See Custom knowledge object coordination for standard mode federated providers.
Write standard mode federated searches
The basic syntax for a standard mode federated search differs depending on the type of remote dataset that you are referencing in the search. A federated search of an events index dataset requires different syntax than a federated search of a metrics index, saved search, last job, or data model dataset.
However, all standard mode federated searches require a reference to at least one federated index that you have defined on your federated search head. This federated index maps to a remote dataset on the federated provider such as an events index, a metrics index, a saved search, a data model, or the last job run by a scheduled search. See Map a federated index to a remote Splunk dataset.
You can use Boolean operators such as AND
and OR
to reference multiple federated indexes in a subsearch.
If your role has the admin_all_objects and indexes_edit capabilities, you can view the federated indexes to which you have access and the remote datasets that those federated indexes map to on the Federated Indexes listing page at Settings > Federated Search > Federated Indexes. If your role does not have this capability, get the names of the federated indexes that you can search from your administrator.
Syntax for standard mode federated searches of remote datasets
The following table provides the search commands and syntax required to search various dataset types on a standard mode federated provider. All of the examples use the federated:
prefix to invoke a federated index on the federated search head that maps to a dataset on the remote search head.
When you run a standard mode federated search, you must follow the syntax for the dataset type to which your federated index is mapped. For example, you cannot use the from
command to reference an events index, and you cannot search a metrics index without using mstats
and referencing the federated index to which the federated index is mapped with a WHERE clause. When your search syntax does not match the dataset type, Splunk software returns an error message.
Remote dataset type | Required syntax | More information |
---|---|---|
events index | Use search to search events index datasets.
|
You can use a wildcard symbol (* ) to reference all federated indexes that map to events indexes:
|
metrics index | Use mstats in conjunction with a WHERE clause to search metrics index datasets.
|
You cannot use a wildcard symbol (* ) to reference all federated indexes that map to metrics indexes.
|
saved search | Use from to reference federated indexes that map to saved search datasets.
|
A saved search dataset is composed of the results of an ad-hoc run of a saved search on the remote search head. For a comparison of saved search and last job datasets, see Map a federated index to a remote Splunk dataset. |
last job | Use from to reference federated indexes that map to saved search datasets.
|
A last job dataset is composed of the results of the last job run by a scheduled search on the remote search head. For a comparison of saved search and last job datasets, see Map a federated index to a remote Splunk dataset. |
data model | Use from to search an unaccelerated remote data model dataset.
Use
|
Run federated searches over lookups
If you are running a standard mode federated search that uses the lookup
command to enrich your results you need to consider whether you want the lookup to be processed on the remote search heads of the federated providers invoked in your search or the federated search head of your local Splunk platform deployment. Federated searches that process the lookup remotely have better overall search performance, but there may be reasons that you would prefer to have the lookup be processed on your local Splunk platform deployment.
In a standard mode federated search, you can force a lookup
command to be processed locally by applying local=true
to it. If you do not set local=true
, under specific conditions the lookup
command will be processed remotely.
If the lookup definition and lookup tables expected by the lookup are not present on the search heads on which it is processed, the lookup search will not return results.
For details, see the lookup
reference topic in the Search Reference.
Restrictions for standard mode federated search
Standard mode federated search does not support the following things:
- Generating commands other than
search
,eventcount
,from
,loadjob
,mstats
, ortstats
. For example, standard mode federated searches cannot include thedatamodel
orinputlookup
commands. You can find a list of generating commands in Command types, in the Search Reference. - Using
from
to reference events index or metrics index datasets. - Real-time search.
- Metrics-specific search commands other than
mstats
, such asmpreview
ormcatalog
. If you must use these commands in a federated search, consider mapping a federated index to a saved search or last job dataset that runs them. See Map a federated index to a remote Splunk dataset.
Standard mode federated search examples
Here are examples of federated searches of remote datasets on standard mode federated providers.
Simple search of a large remote events index dataset
Say you have a federated index on your local search head named airline_flights_SF
. This federated index is mapped to an events index on a remote deployment that is defined as a federated provider for your local deployment. The remote events index contains 2,000,000 events relating to airline flight departures and arrivals at a San Francisco airport. To return the first 100 events of this large remote dataset, run this search:
index = federated:airline_flights_SF | head 100
As a best practice, run a simple search like this on a federated index that you're unfamiliar with. This practice helps you determine whether the federated index name is valid, since you won't see data if the federated index is invalid. It also helps you understand whether the dataset that the federated index is mapped to holds data that is worth searching or correlating with information on your local deployment.
Simple search of a remote metrics dataset
Following on from the previous example, you might also have a federated index on your local search head named airline_flights_SF_metrics
. This federated index is mapped to a metrics index on a remote deployment that is defined as a federated provider for your local deployment. This remote metrics index contains a significantly large number of metric data points that also relate to San Francisco airline flight arrivals and departures.
The following search returns a count of a specific airline departure metric.
| mstats count(SF.airline.splunkpony.departure_delay_time) where index=federated:airline_flights_SF_metrics
For more information, see mstats
in the Search Reference and Overview of metrics in Metrics.
Simple search of a last job dataset
The following search runs over the last job of a scheduled search. The scheduled search runs at the start of each hour and returns information about security badge errors at gates for the preceding hour.
| from federated:badge_errors_last_hour | timechart span=1m count by gate usenull=f
Combine local and standard mode federated search results for comparative analysis
Say you want to investigate the performance of specific airline carriers at competing airports. The following search uses the union
command to combine a remote saved search dataset of San Francisco flight data and a local events index dataset of New York City flight data. Then it finds the average arrival delay between airports in the combined dataset, broken out by airline.
| union
[search index=federated:search_of_airline_flights_SF]
[search index = airlinedata_NYC]
| stats avg (ArrDelay) by UniqueCarrier
If you want to search a local events index and a remote events index, you can use an OR
operator to combine the results:
index = airlinedata_NYC OR index = federated:airline_flights_SF
| stats avg (ArrDelay) by UniqueCarrier
Aggregate results with join for a count of unique values
The following standard mode federated search uses the join
command to aggregate results from the remote and local deployments. It returns the total count of airline carriers running flights out of San Francisco and New York City.
index = federated:airline_flights_SF
| join left = L right = R where L.UniqueCarrier = R.UniqueCarrier
[search index = airlinedata_NYC]
| stats count
Filter a remote data model dataset search by one or more index names
When you run a tstats
search of a remote data model dataset on a standard mode federated provider, it is often helpful to filter the results of the search by one or more indexes. Indexes referenced by the WHERE
keyword must be present on the remote search head.
| tstats count from datamodel = federated:remote_dm_1
WHERE index=index_fs_1 OR index=index_fs_2
You can also run tstats
searches that mix local and federated indexes.
|tstats count where index=local_index_1 OR index=federated:remote_dm_1
Filter a remote data model dataset search on child data model datasets
When you run a tstats
search of a remote data model dataset on a standard mode federated provider, you can filter the results on one or more of the child datasets within the data model by referencing the nodename
of the child dataset.
| tstats count from datamodel = federated:remote_dm_1
WHERE nodename = DM_1_DS.DM_1_DS_CHILD_1
See tstats
in the Search Reference for more information about using nodename
to filter tstats
searches on child data model datasets.
Search over a transparent mode federated provider
When you run searches over a transparent mode federated provider, you can search as if you were searching over your local deployment. No special syntax is required. The search permissions associated with your role govern what you can search on the federated provider.
Searching accelerated data models in transparent mode
After you set up transparent mode federated search, accelerated data models on your local Splunk platform deployment create data model acceleration summaries on indexers of your local deployment and on indexers of your transparent mode federated provider. You do not need to set up accelerated data models on your transparent mode federated provider.
In your federated search, reference a local accelerated data model to return both local and remote results.
The ability to run transparent mode federated searches over accelerated data models requires that both your local and remote Splunk platform deployments be at either Splunk Cloud Platform 9.0.2303 or higher, or Splunk Enterprise 9.1.0 or higher.
See About data models and Accelerate data models in the Knowledge Manager Manual.
Restrictions for transparent mode federated search
Transparent mode federated search does not support the following things:
- Real-time search.
- Using the
meventcollect
command in searches. - Using the
datamodel
command to search remote data models. - Using
from
to search saved search datasets on the federated provider. You can usefrom
to search saved search datasets on your local deployment. - Using the
federated:
syntax to refer to specific federated indexes. Transparent mode searches do not use federated indexes.
Blocked commands for transparent mode federated search
Certain commands are blocked in transparent mode federated searches. When you use a blocked command in a transparent mode federated search, the search returns a warning message and returns results only from your local search head.
The following commands are entirely blocked in transparent mode federated searches:
delete
dump
map
rest
run
runshellscript
script
sendalert
sendemail
.
The commands in this list are classified as risky commands. See SPL safeguards for risky commands in Securing Splunk.
In addition, in certain situations, the makeresults
and tstats
commands are blocked or restricted.
Transparent mode federated searches that use the makeresults
command are blocked unless you use the splunk_server
or splunk_server_group
argument to reference a search head, search head cluster, indexer, or indexer cluster on either your local Splunk platform deployment or the remote federated provider involved in the search. Splunk software processes the makeresults
search on the local or remote server that you indicate with either of these arguments. For more information see makeresults in the Search Reference.
tstats
searches that include a FROM
clause are blocked for transparent mode federated searches over federated providers with Splunk Cloud Platform versions lower than 9.0.2303 or Splunk Enterprise versions lower than 9.1.0. If you use multiple transparent mode federated providers, the tstats
search is processed only on federated providers with qualifying versions. For more information see tstats in the Search Reference.
Troubleshooting federated searches
Federated searches can fail to return events for a variety of reasons. The following table covers the most common error messages and conditions, and gives you some ways to resolve them.
Error message or condition | Resolution | Link to more information |
---|---|---|
Invalid federated index specified | Provide the name of a federated index that exists on your local deployment. | Create a federated index |
Invalid federated provider specified. | Your search references a federated index which specifies a nonexistent federated provider in its definition. Contact your administrator and have them correct the federated index definition. | Create a federated index |
Unauthorized. | The service account credentials that have been set up for a federated provider in this search may be incorrect. Verify that the Service Account Username and Service Account Password that have been designated for the federated provider are correct. | Service accounts and security for Federated Search for Splunk |
If the federated provider uses transparent mode and the service account user for the provider has a role that does not have the fsh_manage capability, searches might not run. | Service accounts and security for Federated Search for Splunk | |
If the federated provider uses standard mode, the service account user for the federated provider should have permission to search the index, saved search, and data model datasets on the federated provider that are involved in the search. For example, if the search references an index dataset on the federated provider, the service account for that federated provider should have the ability to search that index dataset. This ability is associated with the service account role. |
Manage knowledge object permissions in the Knowledge Manager Manual | |
Zero results returned. | If the federated provider uses standard mode, verify that your role's permissions give you access to the federated indexes referenced in the search. | Service accounts and security for Federated Search for Splunk |
If the federated provider uses standard mode, verify that the federated index is mapped to a valid remote dataset. | Create a federated index | |
If the remote dataset you are trying to search is an events index, verify that the events index is not disabled. | If you need to verify whether an index is disabled:
| |
If you are using a standard mode federated provider and the remote dataset you are trying to search is a knowledge object such as a saved search or data model, verify that the knowledge object's permissions enable you to search it. | Manage knowledge object permissions in the Knowledge Manager Manual | |
If possible, run a search job directly on the remote search head to verify that the search head is working correctly. | ||
If you are using a standard mode federated provider, verify that the remote dataset that the federated index maps to contains more than zero events. | ||
Search fails to complete, possibly with the error "Socket error during transaction. Socket error: Success" | Splunk software can terminate federated searches when their search result preview generation duration exceeds a timeout set by another component in your network, such as an elastic load balancer (ELB). When your search is terminated in this manner you can change the max_preview_generation_duration setting in federated.conf to a number of seconds lower than the timeout.For example, if you have an ELB that times out at 60 seconds, you might set the max_preview_generation_duration to 55 .
|
You can change this value with a REST API call. See Federated search endpoint descriptions in the REST API Reference Manual. |
Knowledge object issues for standard mode federated providers | The knowledge objects aren't where the federated provider definition indicates they will be. Verify that the knowledge objects belong to the application identified by Application Short Name, and that the application is installed on the federated provider. | Set the app context for standard mode federated providers |
Your search includes custom knowledge objects that are not duplicated on the remote search head of the federated provider. | Custom knowledge object coordination for standard mode federated providers | |
Verify that the permissions of knowledge objects on the remote search head are set so that users running federated searches from the local search head can access them. | Manage knowledge object permissions in the Knowledge Manager Manual. | |
Knowledge object issues for transparent mode federated providers | Did you run a search within minutes of setting up a transparent mode federated provider? The process by which knowledge objects are transferred from your local deployment to a transparent mode federated provider takes a few minutes. You might be encountering issues because the knowledge objects upon which your search depends aren't yet present on the remote search head. This latency can vary depending on your network bandwidth and the size of the knowledge object bundle. Wait and try your search again. |
Give your users role-based access control of federated indexes | Turn off transparent mode |
This documentation applies to the following versions of Splunk Cloud Platform™: 9.2.2406 (latest FedRAMP release)
Feedback submitted, thanks!